AttentionQwen3_5Full sets the output projection (self.wo) bias based on args.attention_qkv_bias. In the rest of the llama attention implementations, the output projection is bias-free, and HF checkpoints typically don’t include an o_proj.bias. If attention_qkv_bias is enabled, this will introduce an extra bias parameter that won’t be loaded from the checkpoint (and will change numerics). Consider forcing wo to bias=False (or gating it on a dedicated flag rather than attention_qkv_bias).

_recurrent_gated_delta_rule uses a Python for loop over sequence_length, which will be very slow for long prefill sequences and hard for compilers/export backends to optimize. Consider rewriting this recurrence using vectorized/scan-style tensor ops (or constraining this path to decode-only seq_len==1 and providing a separate efficient prefill implementation).

key_dim is computed but never used, which makes the legacy in_proj_qkvz split logic harder to follow. Either remove it or use it to validate the packed tensor layout (e.g., sanity-check expected q/k/v sizing).

AttentionGatedDeltaNet maintains internal conv_state/recurrent_state, but when input_pos is omitted it never resets, so outputs can depend on prior forward calls (state leakage across sequences) when the model is run without kv-cache / without passing input_pos. If this attention is only valid in kv-cache mode, consider asserting input_pos is not None (or alternatively resetting state when input_pos is None) to avoid silently incorrect results.

_load_checkpoint_from_safetensors() loads every shard into shard_to_weights before merging, which can significantly increase peak memory for sharded Qwen3.5 checkpoints. Consider switching to the streaming pattern used in examples/models/qwen2_5/convert_weights.py (group keys per shard, load one shard at a time, then del shard_data) to reduce peak RAM usage.

qwen_3_5_to_meta() currently catches the exception from get_mapped_key(...) and then unconditionally continues for all unmapped keys. This makes conversion silently succeed even if a required text weight is missing from _QWEN_3_5_TO_META (e.g., due to a naming change), which can lead to hard-to-debug runtime failures later. Suggestion: explicitly whitelist/skip known non-text prefixes (visual/MTP/etc.), but for any other unexpected key either raise (default) or at least collect and print a summary of skipped keys behind a --verbose/--strict flag.

The legacy packed tensor handling (*.linear_attn.in_proj_qkvz.weight and *.linear_attn.in_proj_ba.weight splitting) is new behavior but isn’t covered by the added unit test (which only uses the already-split in_proj_qkv/in_proj_z and in_proj_b/in_proj_a keys). Please add a small test case exercising both packed keys and validating the produced in_proj_qkv, in_proj_z, in_proj_b, and in_proj_a outputs (including shape checks).

In qwen_3_5_to_meta, keys that don’t start with model. or lm_head. are currently always skipped (even when they’re not in the explicit ignore list). This can silently drop unexpected checkpoint prefixes and produce a partially-converted/empty state dict without error. Consider raising a ValueError for non-text keys unless _should_ignore_unmapped_key(...) returns true (or at least logging them in non-verbose mode).

The mapping includes model.layers.*.linear_attn.conv1d.bias -> layers.*.attention.conv1d.bias, but the implemented DeltaNet conv (nn.Conv1d(..., bias=False)) has no bias parameter. If HF checkpoints contain this bias, it will become an unexpected key and be ignored by the loader (strict=False), potentially degrading correctness. Either remove/ignore the conv1d.bias mapping or enable a bias in the model and load it consistently.

Legacy packed key support (*.linear_attn.in_proj_qkvz.weight and *.linear_attn.in_proj_ba.weight) is new behavior here, but the unit tests don’t cover these split paths. Adding a small test case that includes packed tensors and asserts the expected split keys/shapes would help prevent regressions.

Hi @Phineas1500 thanks for putting up this PR! I'm wondering if you were able to successfully export to executorch using this config?

Hey! I validated qwen3_5_0_8b with qwen3_5_xnnpack_fp32.yaml, and I exported successfully to .pte. I asked the model what 2+2 was, and it said 4 😂

I also successfully exported qwen3_5_4b with xnnpack not enabled (my MacBook Air isn't powerful enough to fully do it).

I exported qwen3_5_4b with xnnpack successfully on Google Colab with max_seq_length and max_context_length equalling 128, and the resulting qwen3_5_4b_no_backend_smoke.pte was 15.5GB, while the qwen3_5_4b_xnnpack_fp32_128.pte was 18GB.

Currently trying to export qwen3_5_4b with xnnpack and q8da4w so I can try running it on a phone. Hopefully should work 🤞

Note, I was using the code from this PR and from #17801

Wow, that's great to hear! Thank you

tested this exact config for 0.8B, 2B, and 4B, and export to pte was successful. models were able to answer questions.

Great! Also not for this PR but it would be great to add a CI test for the 0.8B version that tests e2e export, lower and run with xnnpack.

nit: could we extract these out into a function, e.g.

def _make_args(self, **kwargs):
          defaults = dict(
              dim=32, n_layers=1, n_heads=4, n_kv_heads=2,
              head_dim=8, hidden_dim=64, max_seq_len=16,
              max_context_len=16,
          )
          defaults.update(kwargs)
          return ModelArgs(**defaults)

Do we still need to register qwen3_5_full? Seems like we can just use mha - and tests use a mix of both.

nit: might be simpler to check the valid ones inline

if not (key.startswith("model.layers.") or key.startswith("model.embed")
          or key.startswith("model.norm") or key.startswith("lm_head")):
      continue

Thanks @Phineas1500 - so I think we can remove _should_ignore_unmapped_key, the function definition and the _IGNORED_UNMAPPED_* items

Ah maybe we can't remove _IGNORED_UNMAPPED_SUFFIXES. That's fine then

Made a commit here and on #17801 addressing your comments!

Is this a real use-case? Are there published checkpoints that use this format or can we drop it (and add it later if it becomes necessary)

Not for this PR but this is copied in each of the qwen directories (and potentially others). We should extract this out into a util in checkpoint.py

This change adds .type_as(x) to the default (non-add_unit_offset) path, which changes behavior for ALL existing models using RMSNorm, not just Qwen3.5. Previously, when x was bfloat16 and self.weight was float32, the multiplication would produce a float32 result. Now the weight is first cast to x's dtype (e.g. bfloat16), keeping the result in that dtype. While this makes the behavior consistent with the add_unit_offset=True branch and may be intentionally fixing dtype propagation, it could subtly change numerical results for existing models using lower-precision dtypes. Please confirm this is intentional and that downstream models have been validated.

This __init__.py uses a lazy __getattr__-based pattern with a dynamically created Qwen3_5Model class, which is inconsistent with every other model package in this codebase (e.g., qwen2_5/__init__.py, qwen3/__init__.py, phi_4_mini/__init__.py, smollm2/__init__.py, gemma/__init__.py, etc.), all of which define the model class at module scope via a direct import. For consistency, consider matching the established pattern:

from executorch.examples.models.llama.model import Llama2Model
from executorch.examples.models.qwen3_5.convert_weights import convert_weights

class Qwen3_5Model(Llama2Model):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)

__all__ = ["Qwen3_5Model", "convert_weights"]

This changes the llama __init__.py from a direct import (from .model import Llama2Model) to a lazy __getattr__ pattern. All other model packages that import Llama2Model (including qwen2_5, qwen3, phi_4_mini, smollm2, gemma, granite, etc.) still use a direct top-level import from executorch.examples.models.llama.model. If there's no circular import or startup performance issue driving this change, it adds unnecessary complexity and inconsistency. Consider keeping the direct import pattern consistent with the rest of the codebase.